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ABSTRACT 

Technology Enhanced Learning (TEL) is a new pedagogical domain aiming to study the usage of 
information and communication technologies to support teaching and learning. The following 
study investigated how this domain is used to increase technical skills in Computer Science (CS). 
A Delphi method was applied, using three-rounds of online survey questions, given to 17 TEL 
experts from different European countries. The results showed that these experts consider TEL 
an effective and interesting support to acquire CS skills. Furthermore, the findings revealed the 
five best tools in TEL to acquire necessary CS knowledge. Future research can provide a 
guideline to implement effective TEL tools in CS studies. 

This manuscript investigates Technology Enhanced Learning (TEL) tools and its applications to 
learn specific skills in CS. As general content the results of a Delphi study that identifies the 
benefits of TEL while teaching are highlighted. 

Keywords: CS studies, Technology Enhanced Learning, learning methods, teaching methods . 


INTRODUCTION 

Despite a growing demand for work in the information technology sector, the number of students 
enrolled in courses which prepare future professionals for CS in Europe is decreasing (OCDE, 
2006). This decline brings as consequence a low number of professionals in the field and further 
on different economic impacts for European countries. 

Many studies have attempted to identify the reasons of this decline, pointing out that CS curricula 
are not always well adapted to industry because it contains many unrelated courses (Kobayashi 
et al, 2008; Gil et al., 2010; Slaten, 2005). Furthermore, the training for skills required to work in 
CS is not properly provided in current educational activities and also that the poor quality of 
teaching is to blame (Plice & Reinig, 2007). 

Because of this two important needs are highlighted: 

It is important to establish a list of unique skills required in order to perform CS work. This list 
must represent the necessary competencies demanded by industry and therefore represent a 
clear expected content in CS curricula. 
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It is also primordial to define dynamic teaching and learning alternatives that can cooperate in the 
acquisition of these skills. 

The following investigation determines if the strategic implementation of TEL can represent an 
effective means to acquire the skills represented in CS curricula. The short list of knowledge and 
skills has been identified in a preliminary research phase as Mathematical Logic, Algorithms, 
Programming, Data Bases, Data modeling and Operative Systems (Porta et al., 2011). This 
preliminary list of skills will also be integrated to this investigation in order to confirm its content 
from the expert’s perspective. 

The main purpose of this research was to recognize the tools and programs available online to 
acquire CS skills and define the position of TEL experts in relation to these tools. This 
investigation uses the Delphi methodology among experts in the TEL field. The experience will 
confirm the list of specific skills in computer science and will make it possible to establish a 
connection between them and the TEL material to learn them. 

The structure of this investigation is presented below: 

First, this article presents previous literacy about TEL and its use to teach CS. 

Secondly it will provide some definitions that clarify CS concepts. 

Next, it explains the Delphi methodology that uses survey rounds to obtain information from a 
panel of experts. This method was applied during this research work. 

Furthermore, the results and findings of the study are presented along with the conclusions in 
each round. 

Finally, the discussions in this study allowed us to offer an overview of the investigation’s results. 


1. BACKGROUND 

The following section provides theory about TEL related to the acquisitions of skills in CS. It will 
also clarify the description of CS itself for the comprehension of the investigation. 


1.1 Technology Enhanced Learning: an effective tool to impede the CS declining 

In 2008, “The Economist journal” published an article entitled “The future of higher education” and 
explain how technology will shape learning in a near future. This investigation demonstrates how 
education needs to evolve towards the usage of technology to be effective. According to Glenn 
(2008), “These changes will have a significant ripple effect on higher education”. The author 
affirms that advanced technologies will reach many more individuals around the world permitting 
specializations in curriculum and teaching methodologies than ever before. “With these benefits 
comes the challenge of ensuring that university infrastructure and operations are in place to 
support the adoption of technology on campus”. This investigation also affirmed that nearly two- 
thirds (63%) of the survey respondents from both the public and private sectors, declare that 
technological innovation will have a major influence on teaching methodologies over the next five 
years. 



48 IJ EDICT 


Technology has become a core differentiator in attracting students and corporate partners. The 
usage of technology material to teach or learn is what we recognize as TEL for the understanding 
of this investigation. The definition of this teaching/learning support is explained below. 

Technology Enhanced Learning (TEL) investigates how information and communication 
technologies can be used to support learning, teaching and competence development throughout 
life of all domains of study. It is first of all a sub-discipline that belongs to the pedagogical 
discipline itself and includes all computer and technological supported collaborative learning 
(Tchounikine et al., 2009). This support retains the specific principles of learning using a 
technology environment. In other words, TEL represents the technological support of any 
pedagogical approach e.g. a toy that teach a baby to give his or her first words; a video game that 
improves logical skills; learning how to play piano using iPad® or computer programs created 
with the unique objective of learning. Hence, all the activities that involve the use of technology 
and that improve the knowledge of a person (user) are considered TEL. 

According to Tchounikine et al. (2009), ‘ Technology-enhanced learning (TEL) is a research arena 
where different disciplines such as computer science, education, psychology, philosophy, 
pedagogy and communication intersect. Along with this description, Tchounikine believes that 
TEL can represent a good alternative to support classical teaching in order to obtain better 
learning results. 

Previous investigations have demonstrated that TEL can be implemented to enhance teaching 
and learning of almost any subject, even domains like Chinese poetry (Cao, 2009) or music 
(Manteley, 1996). 

TEL may also represent an effective alternative to enhance CS education and motivate 
increasing numbers of students to enroll in this degree; therefore, it can impede the declining 
number of young Europeans interested in perusing CS in university. 

There are already some initiatives to attract and retain talent to CS by sharing technology 
experiences (Cusso et al., 2009) and by creating teaching and learning methods to support 
students with the help of information technologies such as “Effective video clips for learning Web- 
languages” such as HTML (Kobayashi et al, 2008); or “Using 3D animation environments to teach 
introductory CS courses” (Cooper, 2003). Consequently, we can assume that CS studies can 
also benefit from teaching methods and alternatives supported by TEL. 

2. What is Computer Science? 

In order to clarify the content in the following investigation the definition of computer science is 
provided in next section of this document. Realizing that CS can have different branches and is 
represented with different names in European curricula we took system engineering (the most 
common field of CS) to perform this study. 


2.1 Definition 

“Computer Science is the systematic study of the feasibility, structure, expression, and 
mechanization of the methodical processes (or algorithms) that underlie the acquisition, 
representation, processing, storage, communication of, and access to information ”( Streubel, 
2011 ). 
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2.2 Computer Science Knowledge and Skills 

In order to understand what is the following investigation intending to do; it is necessary to make 
the difference between knowledge and skills: 

According to Khudkhudia (2011), “Knowledge refers to learning concepts, principles and 
information regarding a particular subject by a person through books and other sources. Skill 
refers to the ability of using that information and applying it in a context”; i.e. knowledge means 
retained theory and skills refers to successfully applying that theory in practice and getting 
expected results. 

From this point of view, this investigation is willing to find out skills (knowledge learned in 
university that will be applied in practice and demanded in industry). 

It is then important to make the difference between two kinds of skills: 

Technical skills (also called directive skills or specific skills) are the ones that determine a 
concrete professional space. They represent the capacity to coordinate the different resources 
intervening in an action (Guedean, 2008). Programming and creating data bases are simple 
examples of technical skills in CS. 

Transversal skills (also known as generic skills) are the common skills between professional 
profiles and disciplines. These skills reflect the ability to solve situations thanks to the knowledge 
we have gained (Guedean, 2008). 

Our investigation refers to technical or specific skills that are represented by different university 
courses in CS programs and that will be the bridge between the gained knowledge and its 
application in the field. 

A preliminary investigation was dedicated to list the unique specific skills in CS (Porta et al., 
2011). The study used a paired analysis comparison from which a first list of skills was obtained. 
The application of this methodology permits to strategically compare CS curricula among 
universities in order to obtain the common subjects. 

Furthermore, the list was compared with programs of other domains of study to obtain a list of 
skills that are unique to CS. The resulted list of skills was finally compared to job offers in the 
market. The common content listed in these curricula represented our point of departure for the 
creation of a former list to be studied. The universities contributing to this finding are: 

Universitat Politecnica de Catalunya, Spain 
Universite d’Evry Val d’Essone, France 
Tallin University of Estonia 
Catholic University of Leuven, Belgium 
University of Reading, United Kingdom 
Heidelberg University, Germany 

Two main reasons drive us to study this institutions, the first one is that personal contacts from 
these universities could help us confirming the information. Secondly, these institutions regroup 
degrees named specifically “Informatics engineering” with the same amount of years to pursue it 
and representing the same degree. 
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According to this strategic comparison between CS curricula; it was possible to obtain a 
preliminary list of skills that belong exclusively to CS presented below: 

Mathematical Logic 
Algorithms 
Programming 
Data Bases 
Data modeling 
Operative systems 

After obtaining this list, the common skills were confronted to job offers in the market in order to 
prove the completeness of it. Also with the intervention of the French official bureau of 
employment that provide us support to rectify the needs in the working field. 

This list is also integrated to the Delphi analysis presented in this investigation in order to be 
confirmed. 


3. MATERIALS AND METHODS 

The method applied for this particular research is called Delphi. The Delphi method consists in 
selecting a panel of experts in the studied field (in our case TEL), and investigates a subject using 
different rounds of questions. The repetitive enquiries are presented to them using different words 
(to discover contradictions of answers in previous rounds and to double check answers) and the 
answer of questions during the first round lead more questions during the subsequent steps and 
so on. The Delphi method has proven to be a relevant and useful tool in information system 
research (Okoli & Pawlowski, 2004). Following to Linstone & Turoff (1975), this research design 
11 may be characterized as a method for structuring a group communication process so that the 
process is effective in allowing a group of individuals, as a whole, to deal with a complex 
problem ”. 

Agreen on this, “One step further” by Landeta (2006), states that “ the Delphi method was 
conceived as a group of techniques whose aim was to obtain the most reliable consensus of 
opinion of a group of experts by means of a series of intensive questionnaires with controlled 
opinion feedback” 

A Delphi study was chosen as a methodology for this research and as a technique for collecting 
opinions of the implicit weaknesses in relying on a single point of view of this new field called 
TEL. Concretely, this research was designed to analyse the application of TEL in CS skills. To 
achieve the main goal of the study, a web-based Delphi method is designed, in order to obtain 
the judgment of well-known experts in this field from different European countries. The Web- 
based survey tool used for this study is called: Polldaddy. The panellists were contact by e-mail 
and invited to visit the web-site and answer. All communications between the experts were 
conducted online so it was an anonymous, personal and confidential survey. 

It is important to explain that some questions demanded to rate specific data. The panellists could 
score given information by expressing their agreement or disagreement by using a point scale. 
This scale was presented as follows: 


5= Strongly Agree (SA) 
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4= Agree (A) 

3= No Answer (N/A) 

2= Disagree (D) 

1= Strongly Disagree (SD) 


Two kinds of answers could result from the three rounds: specific multiple choice answers and 
testimonials from the experts. In the first case, the responses were integrated to a data base and 
tabulated. These answers are represented further in this article by pie charts. In the second case, 
the testimonials of the experts were collected and analyzed and provide as a result a number of 
quality indicators to evaluate TEL tools for CS. 

Panel Description 

The panel selected was composed by 20 experts from 8 European countries (France, Germany, 
Switzerland, Austria, The Netherlands, Estonia, Lithuania and Spain). Some of them were 
members of the European project Stellar (Sustaining Technology Enhanced Learning at a LARge 
scale) and from EaTEL (European Association of Technology Enhanced Learning). The target of 
our research conformed experts in TEL and computer scientists currently working in TEL 
development. In round 1,17 of them answered the questionnaire and all of them have finished 
the three rounds (response rate, 85%). The gender distribution was composed by 3 females (17.7 
%) and 14 males (82.3%). 

Figure 1 illustrates the target description in this investigation. 



Figure 1: Target description in terms of gender, TEL participation, occupation and usage of TEL 
as a professional (in number of participants per domain). 


Surveys rounds: Procedure 

We used three rounds of information and consensus as suggested by Wilhelm (2001) and applied 
also by So & Bonk (2010) as shown in Figure 2. 
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Figure 2: Round descriptions in this investigation. 


4. RESULTS 

The achieved aims in this investigation are represented by different contributions obtained in 
each round. 

4.1 Round 1 

The aim of this round was to identify various emerging issues of CS skills and its support by TEL. 
The first round started on February 3d. 2011 and provided a week to answer. An email attached a 
preliminary table to be completed with the solutions proposed by TEL to acquire/improve the CS 
skills (Mathematical Logic, Algorithms, Programming, Data Bases, Data Modeling and Operative 
Systems). 

Once the experts had finished the first round, their responses were compiled into a completed 
table that will serve for the construction of a TEL catalogue for CS. Two main contributions were 
obtained from this round: 


Inclusion of the skill “Interoperability” 

This skill did not appear in the preliminary list of skills. The reason is that for many years 
Interoperability was considered a sub-subject of other courses. This skill was added by one of the 
experts participating in the investigation. This person considers that the content of the 
communication between systems, networking and Web-services (Interoperability) has become 
much extended in CS content than it was 10 years ago. Therefore, some universities create 
different alternative courses to include this in the program in a complete way. The rest of the 
experts confirmed and agree that interoperability should represent an individual skill in CS, even 
when the name is not mentioned this way in the program. 

Creation of the TEL material catalogue 

The main contribution during the first round consisted in completing the table of available sources 
provided by TEL to learn Mathematical logic, Algorithms, Programming, Data Bases, Data 
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modeling, Operative systems and Interoperability. This table represents a catalogue of TEL 
courses to learn CS. For practical uses of this article, a part of the catalog is illustrated in Figure 
3: 


Programming 


C++ Language Tutorial 1 from its basics up to the 
newest features of ANSI-C++, including basic 
concepts such as arrays or classes and advanced 
concepts such as polymorphism or templates. 
http://www.cplusplus.com/doc/tutorial/ 

HTML. JavaScript, CSS, XML, PHP online course: 
Excellent quality tutorials, dinamic with excersices 
and authocorrections. http://w3schools.com/ 

Delphi: Implementing a SOAP based Web service 
using Delphi tutorial, http://www.xmethods.net 

Java Learning Game: Game that let you learn Java 
coding and running examples with autocorrection 
http://javala.cs.tut.fi/en/ 

Video tutorials: Complete set of videos with 
examples and explanations for Java, C, C++ and 
other languages 

http://www.thenewboston.com/?cat=21&pOpen=tut 

orial 
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Figure 3: Example of the catalog content 


The integral catalogue can be found in Annex A of this document. 


4.2 Round 2 

The second round was conducted in February 16th 2011 and the participants had also a week to 
answer. Four questions were used to create the second round survey. The experts were given 
the questionnaire and asked the rate of fit or agreement of the content of the table product of 
round 1 for each of the seven skills. A 5-point scale was used to rate the items. 

The panellists were also asked to provide comments about their answers. The feedback was 
structured with the experts’ answers, classified and ordered. This survey is available in Annex B 
of this document. 

Represented with a four question survey, this round aloud to understand the expert’s position of 
TEL as learning and teaching support of the skills required to develop CS work. The investigation 
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is developed from the perspective of experts in the field that answer to both TEL and CS know¬ 
how. The relevant results from this round are included here below. 

Q1: How do you agree with the general content of the catalogue (sources provided by TEL, 
round 1)? In order to answer this question, the panelists had access to the TEL catalogue 
obtained in round 1. 



S/\ 

A 

N/A 

D 

SD 

Mathematical Logic 

6% 

70% 

6% 

18% 

0% 

Algorithms 

12% 

65% 

17% 

6% 

0% 

Programming 

23% 

65% 

12% 

0% 

0% 

Data Bases 

29% 

59% 

12% 

0% 

0% 

Interoperability 

18% 

65% 

17% 

0% 

0% 

Data Modelling 

23% 

59% 

18% 

0% 

0% 

Operative Systems 

18% 

65% 

17% 

0% 

0% 


Table 1: Expert’s second round answers concerning the completeness of the CS TEL catalog (in 

percentage per skill). 


The participants were asked whether they agree or not with the general content of the catalogue. 
If the respondent didn’t agree with this content, he or she could specify the reason why and/or 
include extra sources (that have been missed before) in order to complete the catalogue. 

The core of this question is to confront the experts to previous answers in round 1 and eventually 
find contradictions. 

For most of the skills (all of them excluding Mathematical logic and Algorithms), the participants 
mostly agree in the content of the TEL sources presented in the catalogue and none of them 
disagree or strongly disagree. 

This fact confirms the confidence from the part of the panelists in the completeness of the 
catalogue for the some of the skills (Programming, Data bases, Interoperability, Data modeling, 
Operative systems and interoperability). 

Concerning Mathematical logic and Algorithms, it exist a significant percentage of experts that 
disagree with the content of the catalogue (18% and 6% for each skill respectively). 

Question 1 in round 3 of this investigation will permit to clarify this item by asking them the reason 
of this disagreement. 
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Q2: In your opinion, does TEL facilitate learning in CS studies? 



Figure 4: Expert’s perception about TEL to teach CS (in percentage). 


In general, experts in the TEL field “strongly agree” that TEL usage represents a tool that can 
facilitate the acquisitions of CS knowledge (71%). The rest of the participants (29%) “agree” in 
this fact. With this question, we confirm the positive position of the experts that are working in the 
development of this discipline. 

These results, contribute with other investigations aiming to integrate technology into 
methodologies to teach and learn CS (Kobayashi et al., 2009; Kim et al., 2009; Yang and Chen, 
2008). 


Q3: Why do you think TEL facilitates learning CS studies? 


Teaching/ 

Learning 

method 

(results) 

Technology 

contact 

Authoevaluation 

Distance 

learning 

Interactivity 

Reusability 

Time 

10 

3 

2 

1 

1 

1 

1 


Table 2: Expert’s reasons to teach using TEL (in number of answers) 


The addition of the number of answers (19), exceed the number of respondents (17). The reason 
is that this question permitted several answers from the part of the expert. 
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■ Teaching/ Lear ning method 
(results) 

■ Technology contact 


■ Authoevaluation 


■ Distance learning 


■ Interactivity 


■ Reusability 


■ Time 


Figure 5: Expert’s reasons to teach using TEL (in percentage). 


The answers to this question represent success factors in the usage of TEL. Here, the experts 
could specify why they are complementing classical face-to-face teaching with technology 
support and why is this effective. 

A main reason for considering TEL as a facilitator of CS studies, is that this discipline is perceived 
as an effective teaching/learning alternative that reflects good results when implemented (53%). 
“TEL represents a source of knowledge to attire students by following dynamic learning 
methodologies” (from the survey results). 

The second positive aspect is that it improves the contact with technology, as students are 
learning another important skill (computing) at the same time (16%). “It makes the first contact 
between human and technology by teaching computer science and by using technology.” (from 
the survey results). 

Also, it helps students organize themselves and study with their own motivations at their own 
rhythm, since it proposes auto-evaluation methods (11%). “It is very important to have an 
interaction with a system that can give us real time feedback when we are learning and practicing 
new subjects” (from the survey results). 

Other factors such as the re-usability of the material, the usage in distance, the interactivity and 
creativity of the devises and the reduction of the time required to study are mentioned (5% each 
answer). The participants pointed out for example that TEL “is dynamic and in general facilitates 
the comprehension” (from the survey results). 

These answers represent the collection of positive characteristics in a TEL tool to teach or learn 
CS. Hence, it is concluded that, from the experts’ point of view and in order to be effective, all the 
qualities signaled during this investigation are required. 






A Delphi study on Technology Enhanced Learning 57 


Q4: Do you think that TEL facilitates the acquisition of specific CS skills? 



S/\ 

A 

N/A 

D 

SD 

Mathematical Logic 

24% 

52% 

24% 

0% 

0% 

Algorithms 

41% 

41% 

18% 

0% 

0% 

Programming 

59% 

29% 

12% 

0% 

0% 

Data Bases 

47% 

35% 

18% 

0% 

0% 

Interoperability 

41% 

41% 

18% 

0% 

0% 

Data Modelling 

47% 

29% 

24% 

0% 

0% 

Operative Systems 

41% 

35% 

24% 

0% 

0% 


Table 3: Expert’s second round answers concerning their opinion of TEL to teach each CS skill 

(in percentage, per skill). 


Questions 2 and 3 in this round identified the positive position from the part of experts in front of 
TEL to teach CS in general. The respondents agree in the fact that TEL resources can facilitate 
the acquisition of skills related to CS. 

Question 4 intend to identify if experts maintain this position in front of all the skills required to 
perform CS work, or if the respondents sustain any difference between domains. 

As a result, experts affirm that TEL can be used to acquire the identified CS skills. Even with 
subjects like Mathematical Logic and Algorithms in which they considered that the content in the 
catalogue was not complete enough (as stated in question one of this round and further in 
question 1 of round 3). 

The answers to this question allowed us to conclude in their positive position in front of TEL tools 
to learn all the skills representing CS needs in industry. 
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4.3 Round 3 

The process ended on receipts of a third round from the 17 experts on February 23d. 2011. Each 
panellist received a document with the main results and a link to the third questionnaire. 

The third round questionnaire consisted of four questions in order to assess the results obtained 
in previous rounds and eventually find discrepancies between each round of answers. This round 
also permitted to collect testimonials from the participants. This survey is available in Annex C of 
this document. 


Q1: The number of people disagreeing on the support provided by TEL to learn 
Mathematical Logic and Algorithms resulted representative (18% and 6% respectively) to 
our investigation. Why do you think will be the reason? 

To answer this question, the panelists had access to the answers in round number 2 (question 1). 


Not an easy 

measurable 

skill 

It exist many other 
supports (but 1 
don't knowthem) 

It exist many other 
supports (and 1 will 
like to recommend 
some more) 

Other 

answer 

16 

1 

0 

0 


Table 4: Expert’s confrontation of the low confidence of TEL to teach Mathematical logic and 
algorithms (in number of answers). 



■ Because Mathematical logic 
and Algorithms is not an 
easy measurable skill, then 
is a real challenge to TEL 

■ Because it exist many other 
supports to learn 
mathematical logic using 
TEL (but I don't know them) 


Figure 6: Expert’s confrontation of the low confidence of TEL to teach Mathematical logic and 
algorithms (in percentage). 


This question provided the needed information to understand why, during rounds 1 and 2, the 
respondents do not seem to reflect the same degree of confidence in providing TEL sources to 
learn Mathematical Logic and Algorithms (as compared to the rest of the skills listed in the 
catalogue). 
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Some of the participants (6%) estimate that they might be a lot of TEL tools to learn these skills, 
but they are unknown. Therefore, they were not capable to add anything else to the catalogue, 
but they think it should be completed with something else. Mainly, the reason of this sources 
being unknown, is that teachers in this subject need the course to be personalized to their own 
needs and program. 

Most of the answers (94%) pointed that Mathematical Logic and Algorithms are difficult to assess 
with an exact question and answer implemented system. This fact has made the automatic 
evaluation of this two skills challenging to TEL designers, because, as explained further, in a 
complete TEL tool, assessment represent an important part. 


Q2: The reasons of TEL representing a facilitator to learn CS is defined here below. Could 
you rate these reasons from your point of view? 

Answers from question 3 of the second round were provided in order to answer this question. 
This question was answered using a ranking scale. 


Authoevaluati on 

Teaching/Learnin 
g method (results) 

Interacti vity 

Distance Learning 

Technology 

Contact 

Time 

Reusability 

14,57 

11,14 

10,14 

10,00 

9,43 

6,43 

6,19 


Table 5: Expert’s third round answers explaining the positive aspects to use TEL while teaching 
CS (in rating order). 



■ Authoevaluation 

■ Teaching/Learning method 
(results) 

■ Interactivity 

■ Distance Learning 

■ Technology Contact 

■ Time 

■ Reusability 


Figure 7: Expert’s third round answers explaining the positive aspects to use TEL while teaching 
CS 
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Experts rated the reasons of TEL being a useful tool to acquire CS skills. The most important 
reason appears to be that these tools include assessment and self assessment (21%). As 
compared to other teaching and learning methods where students cannot know whether their 
answers are correct or not until the professor asses their work, TEL provides immediate feedback 
that can be useful for the student out of the class room. 

The second reason pointed from the part of experts is that TEL provides good learning results 
(16%). Considering that most of the participants are also teaching some of these skills, this 
answer is considered tested. 

The third position concerns the interactivity and the possibility to work on distance (15% each 
answer). 

Technology contact, time saving and reusability of the material are ranked the last of the list. 
However, they represent as well important success factors in the implementation of TEL, because 
experts pointed them out in round 2. 

Q3: The five more cited sources in the catalogue to learn CS skills are listed below. Could 
you rate these sources from your point of view? 

The regroupation of every tool suggested in a cathalog was analyzed in order to obtein out of it 
the sources that were proposed mostly. Five of them, representing the more recommended by 
the experts are presented to the panellists who will rate them. It is important to mension that 
every tool in this short list already represents complete TEL material, as it was obtained from the 
catalog created by the experts, however, we considered relevant to score them. 

In order to anwser this question, participants had access to the 5 more cited sources on the 
catalogue. 



■ cplusplus.com/doc/tutori 
al/ 

■ ocw.mit.edu 


■ javala.cs.tut.fi/en 


■ w3schools.com 


■ thenewboston.com 


Figure 8: Expert’s best choice of TEL software to teach CS (in %). 
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Table 6, provides a small description of each source and the complete link to it. 


No. 

Descnpti on 

Link 

1 . 

HTML, JavaScript, CSS, XML, PHP online course: Excellent quality 
tutorials, dinamic with excersices and authocorrecti ons. 

htt p://w3schools.com/ 


2. 

Java Learning Game: Game that let you learn Java coding and running 
examples with autocorrecti on. 

htt p://iavala.cs.tut.fi /en/ 


3. 

Video tutorials: Complete set of videos with examples and 
explanati ons for Java, C, C++ and other languages. 

htt p://www.thenewboston.com/?cat=21&pOpen=tutorial 


4. 

C++ Language Tutorial/ from its basics up to the newest features of 
ANSI-C++, including basic concepts such as arrays or classes and 
advanced concepts such as polymorphism or templates. Oriented in a 
practi cal way, with working example programs in all secti ons to start 
practi cing each lesson right away. 

htt p://www.cplusplus.com/doc/tutorial/ 


5. 

MIT OCW: This course is an intensive review of informati on 

technology. It covers topics in soft ware development methods, data 
modeling and databases, applicati on development, Web standards an( 
development, system integrate on, security, and data communicati ons 

htt p://ocw.mit.edu/courses/ 



Table 6: Detail on the expert’s best choice of TEL software to teach CS. 


Question 2 in this round identify and classify quality indicators in TEL tools. In order to confront 
these results with the chosen sources, the criteria listed as a quality indicator to create effective 
TEL material was compared to the list of most successful sources. 

The objective was to confirm the presence of the quality indicators in the sources cited by the 
experts. The analysis of the chosen sources according to the quality indicators is illustrated in 
table 7. 


Tool 

Assesment/ 

Real Skill 

Learn 

progressively 

Structure 

Dynamism / 
moti vati o 

Videos 

Excersice oriented/ 

Autoevaluati on 

Credibility 

w3schools.com 

* 

* 

* 

* 


* 

* 

javala.cs.tut.fi /en 

* 

* 

* 

* 


* 


thenewboston.com 

* 

* 

* 

* 


* 


CpI uspl us. com/doc/tutorial 


* 

* 


* 


* 

ocw.mit.edu 





* 


* 


Table 7: Analysis of the chosen sources to analyze their content. 

As an example of this analysis we confirm the answers for the best rated source (in both the list 
of chosen sources and the quality indicators): 

W3schools.coml (30% of the responses): 

W3Schools provides optimized learning, testing and training of different CS skills. Figure 9 
indicate the list of courses available online. 


1 Copyright 1999-2011 by Refsnes Data. All Rights Reserved. 
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WEB REFERENCES 

■ 

HTML 4.01 

■ CSS 

■ 

HTML5 

■ CSS3 

■ 

XHTML 

■ PHP 

■ 

XML DOM 

■ XSLT 

■ 

JavaScript 

• XPath 

■ 

HTML DOM 

■ XSL-FO 

■ 

■ 

jQuery 

HTML Colors 

■ Color Picker 


Figure 9: Skills available at w3schools.com 

In terms of quality indicators, the learning outcome after having used this method represent real 
skills and competencies, as the system offer online certifications, where you can become licensed 
in these popular web topics. 

The user or student can learn progressively and at his or her own rhythm and the structure of the 
courses are clear, organized and consistent. The software is dynamic, as it aloud the user to 
create code in one part of the screen, while the final results are presented simultaneously in other 
side of the screen. 

The tool doesn’t include any video (the only missing indicator), but is exercise oriented as it 
provides examples that simplifies and improve reading and basic understanding with immediate 
feedback. 

W3schools have credibility in front of teachers and student because it represents a famous online 
standard regulator. 

This analysis confirms the importance of the quality indicators pointed by the experts answering 
to the survey. 

Additionally, the list of most popular cited sources of the catalogue will identify courses in 
Programming, Data Bases and Data Modeling; confirming the fact that experts identify TEL more 
useful for this skills as compared to others. The next question intends to confirm, the difference in 
the progress of these courses development and the rest of the skills. 


Q4: The sources sited before represent tools to learn specific skills (programming, data 
bases, data Modeling). Why would yo say these skills are more developed than others? 
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■ Because they represent 
easy measurable skills. 


■ Because is what computer 
scientists know the most 
(so they recommend it) 


Figure 10: Expert’s third round answers concerning the characteristics of some CS TEL material. 

The success factor of the sources is related once again to the assessment content on them 
(65%), also, because they represent specific skills and learning outcome in CS (35%). It exist 
some relative publications that value the importance of teaching specific skills in CS or to direct 
this education towards a constructive evaluation. (Luxton-Reillya, A. & Dennya, P., 2010). 

DISCUSSION 

This investigation presented a catalogue of the existing tools proposed by TEL in order to acquire 
the skills related to CS. 

There is no possibility of knowing all the tools conceived for CS learning purposes. However, it 
was possible to have a first approach of the tools that are available to learn these skills. It is 
important to mention that the experts targeted in this investigation were in big part computer 
scientists that could answer to both TEL and CS expertise. Also, the fact of including this table 
several times during our investigation, permit to re-confirm the sources from the part of the 
experts contributing to our study. 

As a conclusion, experts in TEL are confident in the fact that this discipline represents a good 
opportunity while teaching CS skills. However, they feel more secure with subjects such as 
Programming or Database than with mathematical logic and algorithms as they represent courses 
that are easy to evaluate with the possibility of exact question and answer systems. 

In general, they consider that TEL is an effective teaching / learning method that improves the 
contact with technology and help students learning with consideration to their own motivations 
and rhythm. Other factors such as the re-usability of the material (it is needed to prepare the 
class just a first time, then the course can be re-used), the fact that you don’t need to be 
physically present (the usage in distance), the interactivity and creativity of the devises and the 
reduction of the time of study required figured between the answers. 

Additionally, a list of quality indicators of TEL material results from the different rounds, the 
sources of the catalog were also scored and compared to these quality indicators and confirms 
that all of them have the needed characteristics to be considered effective. 
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Future work will define the best method between all the sources proposed in the catalog. Thanks 
to this it will be possible to identify and define the best source in TEL to learn CS skills and to 
provide a guideline of effective TEL design. 

Another important approach could be identifying the negative aspects of TEL in order to 
contribute in the improvement of the domain. 
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ANNEXES 

TEL Catalogue supporting System Engineering Skills 

A complete interface regrouping these sources is going to be available soon at: 
http://www.marcelaporta.com/catalog. 

For instance the following schema presents examples of the different TEL tools applied for 
teaching CS skills. These tools include: Games, Videos, Documents, Tutorials and software. 


TEL Supporting Information Systems Engineering Skills 


Data Bases (DB) and Data N 


Tutor Vista/ Math logic problems 
tutorial. This online contains a 
complete Math-logic course from the 
introduction of the subjetc to 
excersises with complete feedback. 
http://www.tutorvista.com/math/math- 
logic-problems-tutorial 


Online CS modules/ Algorithms. Each lesson 
includes a set of review questions which test the 
important concepts from the lesson and provide 
practice problems. 

http://courses.cs.vt.edu/csonline/Algorithms/Lesson 

s/index.html 


Galore: Math and Logic - Math games, 
word problems, and logic puzzles. 
http://www.yarivh.com/cryptopics/inde 
x.htm 


Bot Colony: Algorithms and Networking for 
Computer Games is an important guide to solve the 
algorithmic and networking problems of modem 
commercial computer games. 
http://www.flazx.us/books/3830-Algorithms-and- 
Networking-for-Computer-Games/ 



Video Tutorials: Introduction to Algorithmic 
Complexity http://xoax.net/comp/sci/algorithms/ 


Algorithm. Ptaytot 


C++ Language Tutorial/ from its basics up to the 
newest features of ANSI-C++, including basic 
concepts such as arrays or classes and advanced 
concepts such as polymorphism or templates. 
http://www.cplusplus.com/doc/tutorial/ 


SQL Course: practice what you learn using the on-line SQL interpreter. Immediate results after 
submitting your SQL commands and create unique tables as well as perform selects, inserts, 
updates, deletes, and drops on your tables, http://www.sqlcourse.com/index.html 

SQL online course: Excellent quality tutorials, dinamic with excersices and authocorrections. 
http://w3schools.com/ 


HTML, JavaScript, CSS, XML, PHP online course: 
Excellent quality tutorials, dinamic with excersices 
and authocorrections. http://w3schools.com/ 

Delphi: Implementing a SOAP based Web service 
using Delphi tutorial, http://www.xmethods.net 

Java Learning Game: Game that let you leam Java 
coding and running examples with autocorrection 
http://javala.cs.tut.fi/en/ 


MIT OCW: This course is an intensive review of information technology. It covers topics in 
software development methods, data modeling and databases, application development, Web 
standards and development, system integration, security, and data communications. 
http:.7ocw.mit.edu/courses/civil-and-environmental-engineering/l-264j-database-intemet-and- 
systems-integration-technologies-fall-2006/ 

Oracle 9i AS: Collection of tutorials showing how to use the Oracle 9iweb services platform tools 
Recent updates include a tutorial on Web Services Dynamic invocation and a tutorial on Web 
Services security, http://www.xmethods.net 


Video tutorials: Complete set of videos with 
examples and explanations for Java, C, C++ and 
other languages 

http :.//www.thenew r boston.com/?cat=21 &pOpen=tut 
orial 


Oracle: Oracle University offers thousands of training courses around the world. 
http://education.oracle.com/pls/web_prod-plq-dady db_pages.getpage?page_id=233 

Online book: A online book that teaches SQL and data modeling for the Oracle Database 
http://philip.greenspun.com/sql/ 


Data modeling videos: Instant Access Data Modeling Online Tutorials. Over 6 hours of Online 
Training Videos for just $30 Free Demo and Free Certification, http://apex.vtc.com/data- 
modeling.php 



Tutorial oi rtiguiimmu. ompieAiiy 
http://www.topcoder.com/tc?module=Static&dl=tut 
orial s&d2=complexityl 


XML: Some highly structured text search problems are most efficiently handled by a relational 
database. http://nlp.stanford.edu/IR-booklitmL/htmledition/xml-retrieval-1 .html 

Oracle: The Java Tutorials are practical guides for programmers who want to use the Java 
programming language to create applications, http://download.oracle.com/javase/tutorial/ 


TEL Supporting Information Systems Engineering Skills 


UML Online Course: A complete online course to introduce the Unified Modeling 
Language http://atlas.kennesaw’.edu/~dbraun/csis4650/A&D/UML_tutorial/ 

MIT OCW: The MIT Integrated Global System Modeling (IGSM) framework is used to study 
the causes of global climate change and potential social and environmental consequences. 

The creation and analysis of such frameworks is used in the implementation of policy. (Image 
from the MIT Joint Program on the Science and Policy of Global Change.) Assignments (no 
solutions), http://ocw.mit.edu/courses/engineering-systems-division/esd-864-systems- 
modeling-and-assessment-for-policy- spring-2010/ 


Introuduction to the Unix Operative System: A 
beginners guide to the Unix and Linux operating 
system. Eight simple tutorials which cover the 
basics of UNIX / Linux commands. 
http://info.ee.surrey.ac.uk/Teaching/Unix/ 

Apple Mac OS X: Describes how to use Apple 
Script and Apple event manager in Mac OS X to 
make RPC calls using SOAP and XML-RPC tutorial. 
http://www.xmethods.net 





Tutorial: Introduction c 
http://edn.embarcadero.com/aracie/3i6o3" 


Modeling videos: UML System Modeling Online Tutorials. Over 6 hours of Online Training 
Videos for just $30 Free Demo and Free Certification. http://apex.vtc.com/Unified- 


Apache Axis: Beginner's introduction to Apache Axis tutorial, http://www.xmethods.net 

SOAP Client: Introducing SOAP Client, a C++ Cliento libraty for the Windows platform tutorial, 
published as an article on the server side, showing the developmento a Web service using the 
systinet platform, http://www.xmethods.net 

Networking (for electronic's engineering) offered by Cisco Systems under membership: 
https://leamingnetwork.cisco.com/docs 


Apple programming: 

http://www'.apple.com/search/?q=programming 


> The Cisco Learning Network 


ftM**** fj tr □ 


■ -. 
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Survey used for the investigation: Round TWO 


Q.1 Personal Information * 

Title First Name Last Name 


Q.2 How do you agree with the general content of the table (sources provided by TEL)? * 



Strongly Disagree 

Disagree 

N/A 

Agree 

Strongly Agree 

Mathematical Logic 

© 

© 

© 

© 

© 

Algorithms 

© 

© 

© 

© 

© 

Programming 

© 

© 

© 

© 

© 

Data Bases (DB) and Data Modeling 

© 

© 

© 

© 

© 

Modeling (System Design) 

© 

© 

© 

© 

© 

Operative Systems 

© 

© 

© 

© 

© 

Interoperability 

© 

© 

© 

© 

© 


Q.3 In your opinion, does TEL facilitate Computer Science studies? * 

© Strongly Disagree 
© Disagree 
© N/A 
© Agree 
© Strongly Agree 

Please help us understand why you selected this answer 


Q.4 After your experience what are the main obstacles presented to the development of 
TEL?* 


Q.5 In your opinion, what are the critical success factors of TEL implementation? * 




Q.6 Do you think that TEL facilitate the acquisition of Informatics' engineering skills? * 



Strongly Disagree 

Disagree 

N/A 

Agree 

Strongly Agree 

Mathematical Logic 

a 

a 

a 

a 

a 

Algorithms 

a 

a 

a 

a 

a 

Programming 

a 

a 

a 

a 

a 

Data Bases (DB) and Data Modeling 

a 

a 

a 

a 

a 

Question Modeling (System Design) 

a 

a 

a 

a 

a 

Operative Systems 

a 

a 

a 

a 

a 

Interoperability 

a 

a 

a 

a 

a 


Q.7 In your opinion which of the sources sited in the table represent the THREE best TEL 
source in mater of learning method and why? * 

What method will you recommend the most. 


^ 

Finish Survey 
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Survey used for the investigation: Round THREE 


Q.1 The number of people disagreeing on the support provided by TEL to learn 

Mathematical Logic resulted representative (18%) to our investigation. Why do you 
think will be the reason? 

Please use the support presentation, slide 2 to answer this question, (choose one or several answers) 

0 Because mathematical logic is not an easy measurable skill, then is a real challenge to TEL 
0 Because it exist many other supports to learn mathematical logic using TEL (but I don't know them) 

□ Because it exist many other supports to learn mathematical logic using TEL (and I will like to recommend 
some more) 

Other 1 | 

Please help us understand why you selected this answers: 


Q.2 The number of people disagreeing in the support provided by TEL to learn Algorithms 
resulted representative (6%) to our investigation. Why do you think will be the 
reason? 

Please use the support presentation, slide 3 to answer this question, (choose one or several answers) 

□ Because Algorithms is not an easy measurable skill, then is a real challenge to TEL 

□ Because it exist many other supports to learn mathematical logic using TEL (but I don't know them) 

□ Because it exist many other supports to learn mathematical logic using TEL (and I will like to recommend 
some more) 

Other | | 

Please help us understand why you selected this answers: 


Q.3 

The reasons of TEL representing a good learning method to learn computer science 
is defined here below in order of importance. Could you rate these reasons from your 


point of view? 









Please use the support presentation, slide 4 to answer this question. (7 represents your best choice) 





1 

2 

3 

4 

5 

6 

7 


It provides good learning results 

© 

© 

© 

© 

© 

© 

© 


It increase the contact with technology 

© 

© 

© 

© 

© 

© 

© 


It includes assessment and self assessment 

© 

© 

© 

© 

© 

© 

© 


Improves distance learning 

© 

© 

© 

© 

© 

© 

© 


Is more interactive than regular methods 

© 

© 

© 

© 

© 

© 

© 


It can be used several times (the needed number of times) 

© 

© 

© 

© 

© 

© 

© 


It saves times 

© 

© 

© 

© 

© 

© 

© 

Q.4 

The main obstacles identified by TEL users (experts) are defined here below in order 


of importance. Could you rate these reasons from your point of view? 




Please use the support presentation, slide 5 to answer this question. (8 represents your best choice) 





1 

2 

3 

4 

5 6 

7 

8 


People does not know/understand TEL 

© 

© 

© 

© 

© © 

© 

© 


You need a lot of money to develop TEL 

© 

© 

© 

© 

© © 

© 

© 


The main obstacle of TEL is related to educational issues (not 

applicable for every domain of study) 

© 

© 

© 

© 

© © 

© 

© 


You need to know a lot about technology to be capable to use it 

© 

© 

© 

© 

© © 

© 

© 


Professors does not consider important to teach using TEL 

© 

© 

© 

© 

© © 

© 

© 


TEL community is missing synergy and group work 

© 

© 


© 

© © 

© 

© 


People have problems adopting new things 

© 

© 

© 

© 

© © 

© 

© 


You need a lot of time to develop in TEL 

© 

© 

© 

© 

© © 

© 

© 


Q.5 The critical success factors of TEL are defined here below in order of importance. 
Could you rate these reasons from your point of view? 

Please use the support presentation, slide 6 to answer this question. (9 represents your best choice) 



1 

2 

3 

4 

5 

6 

7 

8 

9 

It provide good learning results 

© 

© 

© 

© 

© 

© 

© 

© 

© 

It enhance the contact with technology 

© 

© 

© 

© 

© 

© 

© 

© 

© 

It is dynamic 

© 

© 

© 

© 

© 

© 

© 

© 

© 

It is useful to teach 

© 

© 

© 

© 

© 

© 

© 

© 

© 

It save time 

© 

© 

© 

© 

© 

© 

© 

© 

© 

It promotes the creation of learning material 

© 

© 

© 

© 

© 

© 

© 

© 

© 

It can be used several times 

© 

© 

© 

© 

© 

© 

© 

© 

© 

It improves distance learning 

© 

© 

© 

© 

© 

© 

© 

© 

© 

It save money 

© 

© 

© 

© 

© 

© 

© 

© 

© 

Q.6 The five more successful sources of TEL to learn computer science skills are listed 

below. Could you rate these reasons from your point of view? 





Please use the support presentation, slide 7 to answer this question. 5 represent your best choice. 






1 


2 

3 


4 


5 

w3schoob.com 


© 


© 

© 


© 


© 

javab.cs.tut.fi/en/ 


© 


© 

© 


© 


© 

thenewboston.com 


© 


© 

© 


© 


© 

cplusplus.com/doc/tutorial/ 


© 


© 

© 


© 


© 

ocw.nrit.edu 


© 


© 

© 


© 


© 


Q.7 The sources sited before represent tools to learn specific skills (programming, data 
bases, data modelling). Why would yo say these skills are more developed than 
others? 

Please use the support presentation, slide 8 to answer this question. 

© Because they represent easy measurable skills. 

© Because is what computer scientists know the most (so they recommend it) 

© Other 1 | 

Please help us understand why you selected this answer 

I 1 

Q.8 The qualities needed to create effective TEL material to learn computer science skills 
are listed below. How do you agree in terms of importance? 

Please use the support presentation, slide 9 to answer this question. 7 represents your best choice. 



1 

2 

3 

4 

5 

6 

7 

programs that provide self evaluation and online assesment 

© 

© 

© 

© 

© 

© 

© 

dynamic / motivational programs 

© 

© 

© 

© 

© 

© 

© 

videos 

© 

© 

© 

© 

© 

© 

© 

programs that represent real assessed skilb 

© 

© 

© 

© 

© 

© 

© 

programs with good teaming structures 

© 

© 

© 

© 

© 

© 

© 

programs that aloud you to learn progressively 

© 

© 

© 

© 

© 

© 

© 

programs that are supported by a credible institution 

© 

© 

© 

© 

© 

© 

© 


Q.9 What do you think is the future for TEL in the development of computer science 
skills? 


A 


Finish Survey 
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